Applying Data Mining Techniques in Text Analysis

A number of recent data mining techniques have been targeted especially for the analysis of sequential data Traditional examples of sequential data involve telecom munication alarms Www log les user action registration for Hci studies or any other series of events consisting of an event type and a time of occurrence Text can also be seen as sequential data in many respects similar to the data collected by sensors or other observation systems Traditionally texts have been analysed using various information retrieval related methods such as full text ana lysis and natural language processing However only few examples of data mining in text particularly in full text are available In this paper we show that general data mining methods are applicable to text analysis tasks under certain conditions Moreover we present a general framework for text mining The framework follows the general Kdd process thus containing steps from preprocessing to the utilization of the results The data mining method that we apply is based on generalized episodes and episode rules We consider preprocessing of the text to be essential in text mining by shifting the focus in the preprocessing phase data mining can be used to obtain results for various purposes We give concrete examples of how to preprocess texts based on the intended use of the discovered results and how to balance preprocessing with post processing We also present example applications including search for key words key phrases and other co occurring words e g collocations and generalized concordances These applications are both common and relevant tasks in information retrieval and natural language processing We also present results from real life data experiments to show that our approach is applicable in practice